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Abstract 

In  this  paper  we  present  Preventive  Spatio-Temporal  Aggregation  (PreSTA),  a  reputation  model 
that  combines  spatial  and  temporal  features  to  produce  values  that  are  behavior  predictive  and  useful  in 
partial-knowledge  situations.  To  evaluate  its  effectiveness,  we  applied  PreSTA  in  the  domain  of  spam 
detection.  Studying  the  temporal  properties  of  IP  blacklists,  we  found  that  25%  of  IP  addresses  once 
listed  on  a  blacklist  were  re-listed  within  10  days.  Further,  during  our  evaluation  period  over  45%  of 
IPs  de-listed  were  re-listed.  By  using  the  IP  address  assignment  hierarchy  to  define  spatial  groupings 
and  leveraging  these  temporal  statistics,  PreSTA  produces  reputation  values  that  correctly  classify  up 
to  50%  of  spam  email  not  identified  by  blacklists  alone,  while  maintaining  low  false-positive  rates. 
When  used  in  conjunction  with  blacklists,  an  average  of  93%  of  spam  emails  are  identified,  and  we 
find  the  system  is  consistent  in  maintaining  this  blockage  rate  even  during  periods  of  decreased  blacklist 
performance.  PreSTA  spam  filtering  can  be  employed  as  an  intermediate  filter  (perhaps  in-network) 
prior  to  context-based  analysis.  Further,  our  spam  detection  system  is  scalable;  computation  can  occur 
in  near  real-time  and  over  500,000  emails  can  be  scored  an  hour. 


1  Introduction 

Roughly  90%  of  the  total  volume  of  email  on  the  Internet  is  considered  spam  [5],  and  IP-based  blacklisting 
has  become  a  standard  tool  in  fighting  such  influxes.  The  vast  majority  of  spam  originates  from  botnets, 
centrally  controlled  organizations  of  infected  hosts.  Spammers  can  vary  which  hosts  within  a  botnet  are 
sending  spam,  migrating  as  hosts  become  blacklisted.  As  a  result,  some  20%  of  spam  emails  received  at  a 
large  spam  trap  in  2006  were  not  listed  on  any  blacklist  [25]. 

Despite  the  dynamic  nature  of  spamming  IP  addresses,  they  still  exhibit  interesting  spatial  properties. 
In  particular,  the  majority  of  such  IPs  are  clustered  throughout  the  address  space  [25];  Spammers  tend 
to  be  spatially  found  “near”  other  spammers  {e.g.,  within  the  same  subnet  or  Autonomous  System  (AS)). 
This  phenomenon  was  recently  reinforced  when  in  December  of  2008  the  McColo  ISP  was  shutdown  by 
authorities  and  spam  rates  dropped  by  nearly  15%  [5,  21].  Further,  previous  studies  have  demonstrated  that 
the  AS -level  membership  of  spamming  IPs  can  be  used  to  effectively  classify  spam  [15]. 

Temporal  properties  of  spam  originating  IPs,  however,  have  not  been  studied  as  a  measure  of  sender 
reputation.  The  history  of  a  single  IP’s  entry  and  exit  from  a  blacklist  is  significant.  More  crucially,  the 
listing  and  de-listing  patterns  of  spatially  related  IP  addresses  over  time  is  behavior  predictive.  For  example, 
suppose  a  subnet  has  a  dense  blacklist  history.  If  half  of  the  subnet  is  currently  or  was  recently  blacklisted,  it 
would  be  prudent  to  view  the  other  half  suspiciously.  In  our  analysis,  we  found  that  previous  blacklistings  are 
predictive  of  future  listings.  More  than  25%  of  IPs  once  listed  were  re-listed  within  10  days,  and  45%  were 
re-listed  during  our  experiment  period.  Given  these  statistics,  it  becomes  clear  that  the  usage  of  historical 
data,  as  opposed  to  the  static  view  blacklists  provide,  may  be  most  helpful  when  valuating  unknown  entities. 

‘This  research  was  supported  in  part  by  ONR  MURI  N00014-07-1-0907.  POC:  Insup  Lee,  lee@cis.upenn.edu 
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Further,  we  expand  previous  work  regarding  spatial  groupings  of  spam  originating  IP  addresses  by  using 
knowledge  of  the  IP  assignment  hierarchy.  We  found  that  calculating  reputation  at  multiple  levels  provides 
greater  insight  into  unknown  quantities.  For  example,  the  clustering  of  spamming  IPs  within  an  AS  are 
distributed  non-uniformly,  just  as  malignant  ASes  are  found  non-uniformly  across  the  address  space.  We 
find  a  combination  of  both  fine  and  broad  groupings  best  correlate  negative  feedback. 

We  have  developed  a  novel  reputation  model,  PreSTA  (Preventive  Spatio-Temporal  Aggregation), 
that  computes  values  that  are  behavior  predictive  of  unknown  entities  in  partial-knowledge  situations  where 
entity-specific  data  may  be  incomplete.  PreSTA  aggregates  a  history  of  negative  feedback  from  a  database 
(i.e. ,  a  blacklist)  into  reputations  for  entities  based  on  their  individual  behavior  and  that  of  their  spatial  group¬ 
ings.  PreSTA  requires  only  that,  (1)  there  exists  a  grouping  function  to  define  finite  sets  of  participants,  and 
(2)  there  is  observable  negative  feedback  to  construct  a  behavior  history.  For  each  spatial  grouping,  a  sepa¬ 
rate  reputation  value  is  produced  and  machine  learning  can  be  applied  to  tune  the  combination  to  generate  a 
higher-order  reputation.  This  value  can  incorporate  as  many  spatial  and  temporal  dimensions  as  necessary 
to  best  represent  the  entities  being  valuated. 

Spam  is  a  motivating  model  for  PreSTA,  and  we  use  it  to  analyze  five  months  of  mail  logs  provided 
by  our  university  mail  system.  We  do  this  by  simulating  a  production  email  server  implementing  PreSTA, 
complete  with  caching  and  periodic  re-training.  Empirical  results  show  that  PreSTA  can  identify  up  to 
50%  of  spam  emails  not  caught  by  blacklists  while  maintaining  low  false-positive  rates.  When  used  in 
combination,  PreSTA  and  classic  blacklists  are  able  to  consistently  identify  93%  of  spam  without  text- 
based  analysis.  Moreover,  these  blockage  rates  are  stable,  remaining  steady-state  even  as  the  performance  of 
the  underlying  blacklist  suffers.  Further,  this  implementation  is  scalable,  scoring  at  latencies  unnoticeable 
to  an  end  user,  while  valuating  over  500,000  emails/hour.  We  do  not  propose  that  PreSTA  can  replace 
context-based  filtering,  but  instead  may  be  used  as  an  intermediate  filter  (perhaps  in-network),  reducing  the 
number  of  emails  that  require  complex  text-processing  and  the  exposure  of  private  email  content. 

However,  PreSTA  defines  a  general  model  for  spatio-temporal  reputation  which  is  broader  than  spam 
and  IP  blacklists  alone.  Any  application  that  meets  the  two  usage  criteria  of  PreSTA  can  benefit.  Prelimi¬ 
nary  work  [32]  has  already  shown  PreSTA  reputation  values  helpful  in  detecting  vandalism  on  Wikipedia. 
More  generally,  PreSTA  may  be  applicable  to  the  entire  class  of  dynamic  trust  management  systems  [10, 
31],  characterized  by  the  need  for  decision-making  in  the  presence  of  uncertainty.  The  Wikipedia  use-case 
and  other  potential  applications  are  given  greater  attention  in  Sec.  8. 

2  Related  Work 

Previous  work  within  the  spam  detection  community  has  focused  on  using  network-level  properties  to  dis¬ 
tinguish  legitimate  and  malicious  mail.  Unlike  content-based  filters  that  employ  Bayesian  quantifiers  [28], 
these  techniques  leverage  the  reputation  of  the  sender  -  often  the  IP  address  of  the  connecting  mail  server. 

IP'  blacklists  [3],  such  as  Spamhaus  [7],  collate  IP  addresses  of  known  spamming  hosts  based  on  feed¬ 
back  from  varied  institutions  (large  email  providers).  Over  time,  IP  addresses  may  be  listed,  de-listed,  and 
re-listed.  It  is  this  dynamic  nature  of  blacklists  that  PreSTA  leverages. 

Recent  work  suggests  that  blacklists  are  too  latent  in  their  listing  of  new  spammers  [24],  and  that  10% 
of  spamming  IPs  have  never  been  seen  previously  [27].  Even  so,  studies  have  shown  that  the  spamming  IPs 
are  distributed  non-uniformly  throughout  the  address  space.  Eurther,  a  large  percentage  of  spam  originates 
from  small  regions  of  the  address  space  [25].  Generally,  spammers  are  found  adjacent  to  other  spammers, 
and  this  principle  was  shown  to  be  an  effective  metric  in  detecting  new  spammers  by  Hao  et  al.  [15]  in  their 

*IP  blacklists  are  sometimes  referred  to  as  DNS  blacklists,  due  to  the  frequent  use  of  the  DNS  protocol  in  performing  lookups 
(DNS  lookup  is  used  by  mail  servers  that  choose  not  to  store  a  local  copy  of  the  blacklist).  These  lists  contain  only  IPs. 
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development  of  the  SNARE  system.  In  particular,  Hao  et  al.  showed  the  AS -membership  of  an  IP  address  to 
be  relevant.  We  build  upon  this  research,  grouping  IPs  at  multiple  levels  of  the  address  assignment  hierarchy. 

In  combination  with  spatial  measurement,  SNARE  also  utilizes  simple  temporal  metrics  to  perform  spam 
filtering  (for  example,  the  time-of-day  an  email  was  sent),  and  apply  a  lightweight  form  of  aggregation  {i.e., 
mean  and  variance)  over  such  features  to  detect  abnormal  patterns.  In  contrast,  PreSTA’s  temporal  aspect 
is  more  complex,  aggregating  time-decayed  behavioral  observations  that  encode  months  of  detailed  history. 
Indeed,  [15]  identifies  many  valid  measures  of  spamming  behavior,  buf  is  ulfimately  incapable  of  Infernet- 
wide  scalabilify  given  fheir  reliance  on  high-dimensional  learning.  PreSTA  spam  defection  compufes  over 
a  single  feafure,  IP  address  (and  groups  fhereof),  and  is  effective  while  realizing  wide-scalabilify. 

Oufside  of  academia,  several  commercial  services  claim  fhe  use  of  fechniques  similar  fo  fhose  proposed 
herein.  Eor  example,  Symanfec  [30]  allows  one  fo  look-up  “IP  repufafion”,  a  feafure  used  in  ifs  securify 
soffware.  Eor  fhe  public-facing  service,  fhe  response  fo  queries  is  binary  -  alfhough  we  found  if  correlafed 
well  wifh  fhe  values  PreSTA  calculafes.  In  addition,  fhe  SenderBase  [16]  sysfem  uses  spatial  dafa  fo  build 
repufafions.  Whafever  fhe  infernals  of  such  producfs,  we  believe  fhe  PreSTA  sysfem  makes  a  confribufion 
by  infroducing  fhis  fechnology  info  a  non-propriefary  seffing. 

PreSTA  can  also  be  examined  in  fhe  contexf  of  general-purpose  repufafion  systems/logics,  such  as 
EigenTrusf  [20]  or  TNA-SE  [18].  The  primary  distinguishing  factor  befween  fhese  systems  and  PreSTA 
is  fhe  nafure  of  feedback.  Conventional  algorifhms  usually  aggregate  bofh  positive  and  negafive  feed¬ 
back,  where  feedbacks  are  perpefually  refained  and  associafed  fo  a  single  discrete  fime-sfamp.  In  confrasf, 
PreSTA  considers  only  negafive  feedbacks,  and  funcfions  besf  wifh  expiring  feedback,  where  a  behavioral 
observafion  is  valid  for  some  finite  duration  and  fhen  discarded.  A  fransformafion  can  be  applied  so  PreSTA 
can  operate  using  more  conventional  feedback,  a  fopic  discussed  furfher  in  Sec.  3.2. 

Einally,  mosf  repufafion  sysfems  are  designed  from  a  disfribufed  perspective,  concenfrafing  on  how  un- 
frusfed  parties  can  identify  reliable  partners  in  a  network  setting.  In  our  application,  the  blacklist  provider 
renders  a  singular  and  fully-trusted  perspective.  Moreover,  while  distributed  reputation-network  approaches 
to  spam-filtering  [9,  13]  exist,  these  focus  on  sharing  classifications  rather  than  the  discovery  of  new  ones. 

3  PreSTA  Reputation  Model 

Summarily,  the  reputation  model  begins  by  mapping  negative  feedback  to  some  identifier  (i.e.,  a  user). 
Poor  repufafions  correspond  fo  fhose  who  have  recenfly  commiffed  bad  behavior  (resulfing  in  a  negafive 
feedback).  In  fhe  absence  of  bad  behavior,  repufafions  improve  over  time.  Critically,  user-level  repufafions 
may  also  be  combined  according  fo  fhe  spatial  relafionships  of  fhe  associafed  enfifies. 

We  consider  a  dafabase  where  an  enfify  is  considered  active  when  if  receives  negafive  feedback  (becomes 
listed)  and  inactive  when  fhaf  negafive  feedback  expires  (becomes  de-lisfed).  Eurfher,  one  is  able  fo  query 
fhis  dafabase  fo  see  an  enfify’s  lisfing  hisfory.  When  an  enfify  has  recenfly  been  assigned  a  negafive  feedback, 
fhe  penalfy  associafed  wifh  fhe  lisfing  is  a  heavy  one.  However,  fhe  more  fime  fhaf  passes,  fhe  less  weight 
a  prior  listing  carries.  When  an  entity  is  re-listed  (re-enters  the  database  after  removal),  the  weight  of  the 
previous  negative  activity  is  still  applied,  but  there  is  now  compounding  evidence  against  the  entity  being 
valuated.  These  characteristics  are  well  represented  in  PreSTA  where  each  listing,  re-listing,  and  de-listing 
carries  a  weight  based  on  its  temporal  relevance,  realized  via  a  decay  function. 

Spatial  properties  are  considered  based  on  grouping  functions  which  take  an  entity  as  input  and  return 
all  entities  that  are  a  member  of  the  input’s  group.  We  expect  more  than  one  grouping  function  will  be  given, 
and  an  entity  will  populate  multiple  groups  -  one  for  each  grouping  function.  This  is  advantageous;  in  the 
absence  of  entity-specific  dafa,  spatial  (group)  dafa  is  helpful  in  characterizing  an  enfify’s  expecfed  behavior. 
A  repufafion  can  fhen  be  compufed  af  each  grouping  and  combined  according  fo  application-specific  criteria. 
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3.1  Reputation  Computation 

The  goal  of  the  reputation  computation  is  to  produce  a  quantified  value  that  captures  both  the  spatial  and 
temporal  properties  of  the  entity  being  valuated.  Spatially,  the  size  of  the  grouping  must  be  considered,  and 
temporally,  the  history  of  negative  feedback  must  be  weighted  proportional  to  its  spatial  proximity. 

For  example,  suppose  we  are  valuating  the  group-level  reputation  of  an  entity.  If  only  a  small  portion 
of  members  have  recent  listing  activity,  then  the  reputation  should  be  relatively  high.  Similarly,  if  a  large 
portion  of  group  members  have  a  rich  listing  history,  but  in  the  distant  past,  the  reputation  should  still  be 
high.  However,  if  many  members  have  recent  listing  activity,  the  reputation  should  be  relatively  low. 

To  capture  these  properties,  three  functions  are  required  -  two  temporal  and  one  spatial: 

•  hist{a,  G,  H)  is  a  temporal  function  returning  a  list  of  pairs,  {tin,  tout),  representing  all  entries/exits 
from  the  feedback  history,  H,  according  to  the  grouping  of  entity  a  by  grouping  function  G.  The 
values  tin  and  tout  are  time-stamps  bounding  the  listing.  Active  listings  return  {tin,  -L). 

•  decay{touu  h)  is  a  temporal  function  that  exponentially  decays  input  times  using  a  half-life  h,  and 

it  takes  the  form  where  Af  =  tnow  —  tout  are  of  the  same  unit  as  h.  It  returns  a  value  in  the 

range  [0, 1],  and  for  consistency,  decay{J-,  h)  =  1. 

•  size{cx,  G,  t)  is  a  spatial  function  returning  the  magnitude,  at  time  t,  of  the  grouping  defined  by  G, 
of  which  a  is/was  a  member.  If  G  defines  multiple  groupings  for  an  entity  a,  only  the  magnitude  of 
one  grouping  is  returned.  The  choice  of  which  group  is  application  specific. 

We  now  define  fhe  raw  reputation  function  as  follows: 


rawjrep{a,  G,  H) 


E 

hist{a,G,,H) 


decay  {tout,  h) 
size{a,G,tin) 


(1) 


This  computation  captures  precisely  the  spatio-temporal  properties  required  by  PreSTA.  Temporally,  the 
entry/exit  history  of  the  database  is  captured  at  each  summation  via  the  hist{)  function,  and  events  occurring 
recently  are  more  strongly  weighted  via  the  decay{)  function.  Spatially,  grouping  function  G  defines  the 
group  membership,  and  each  summation  is  normalized  by  the  group  size. 

When  two  or  more  grouping  functions  are  defined  over  the  entities,  multiple  computations  of  rawjrep{) 
are  performed.  Each  value  encodes  the  reputation  of  an  entity  when  considered  within  a  different  spatial 
context.  How  to  best  combine  reputation  is  an  application  specific  decision,  and  for  our  spam  application, 
machine  learning  techniques  are  used  (see  Sec.  5.7). 

The  values  returned  by  rawjrepQ  are  strictly  comparable  for  all  spatial  groupings  defined  by  G  and  the 
history  H.  High  values  correspond  to  less  reputable  entities  and  vice-versa.  However,  it  is  more  typical  for 
reputation  systems  [18,  20]  to  normalize  values  onto  the  interval  [0, 1]  where  lower  values  correspond  to  low 
reputation  and  vice-versa.  Ultimately,  our  machine  leai'ning  technique  does  not  require  normalized  values. 
Such  values  do,  however,  enable  our  model  to  be  consistent  with  other  reputation  systems  and  provide  an 
absolute  interpretation  that  permits  manually-authored  policies  {e.g.,  allow  access  where  reputation  >  0.8). 

Normalization  requires  knowledge  of  an  upper  bound  on  the  values  returned  by  rawjrepQ.  This  cannot 
be  generally  defined  when  the  de-listing  policy  is  non-regular.  However,  if  it  is  known  that  listings  expire 
from  the  negative  feedback  database  after  a  fixed  duration  d,  and  that  listings  are  non-overlapping,  then  it  is 
possible  to  compute  an  upper  bound.  Such  a  bound  can  be  found  by  considering  an  entity  who  is  as  bad  as 
possible;  one  that  is  re-listed  immediately  after  every  de-listing,  and  thus,  is  always  active  in  the  feedback 
database.  Considering  a  grouping  of  size  1,  the  rawjrepQ  computation  reduces  to  a  geometric  sequence: 


4 


(2) 


1 

MAX_REP  =  IH - ttt 

Similarly,  the  same  worst  case  reputation  occurs  for  groups  of  larger  size,  however,  instead  of  a  single 
entity  acting  as  a  bad  as  possible,  the  entire  group  is  simultaneously  re-listed  immediately  following  each 
de-listing.  We  can  now  define  a  normalized  reputation  as: 


rep{a,  G,  H)  =  1  — 


rawjrep{a,  G,  H) 
MAX_REP 


(3) 


Clearly,  small  d  produce  larger  MAX_REP  values.  Thus,  a  precise  value  for  d  need  not  be  known  -  only 
a  lower-bound.  One  may  ask,  “Why  not  choose  an  arbitrarily  small  d  (i.e.,  1  second)  as  the  lower  bound?” 
Such  a  small  lower  bound  will  increase  the  normalization  factor  and  the  distance  between  what  can  be 
considered  good  and  bad  reputation  will  become  arbitrarily  small.  The  best  choice  for  easy  differentiation 
is  the  greatest  lower-bound  of  the  listing  interval.  We  found  that  a  good  estimate  for  d  is  sufficient,  at  least 
in  the  domain  of  spam  detection,  because  the  worst-case  reputation  is  unlikely  to  be  realized. 

The  reputation  computation  defined  herein  can  be  further  specialized  depending  on  fhe  enfifies  being 
valuated  or  fhe  nafure  of  fhe  negative  feedback  database.  For  example,  one  can  eliminate  all  spatial  relevance 
by  using  grouping  functions  that  define  groups  of  size  1.  Or,  one  can  eliminate  all  temporal  calculation 
by  defining  fhe  refurn  of  decay{)  as  a  consfanf  (C).  Bofh  such  usages  are  employed  in  spam  defection; 
fhe  former  due  fo  dynamism  in  IP  address  assignmenf,  and  fhe  laffer  due  fo  properties  of  fhe  blacklisf  in 
question.  Nofe  fhaf  when  decay{tout,  h)  =  C,  MAX_REP  =  decay{±,  h)  +  C. 


3.2  Reputation  Database 

The  repufafion  database,  H,  depends  on  the  nature  of  feedback  available.  PreSTA  is  most  adept  at  handling 
expiring  feedback  like  that  present  in  IP  blacklists.  By  definition,  an  expiring  feedback  occurs  when  an  entity 
is  active  (listed)  in  the  database  before  being  removed  (de-listed)  after  some  finite  duration.  In  this  case,  H 
is  a  record  of  the  entries/exits  of  listings  such  that  the  active  database  can  be  reproduced  at  any  point  in  time. 

Feedback  can  also  be  discrete,  where  negative  feedbacks  are  associated  with  a  single  time-stamp  but 
stored  for  perpetuity.  This  is  the  model  most  frequently  seen  in  general-purpose  reputation  management 
systems  [18,  20].  In  such  cases,  hist{)  always  returns  pairs  of  the  form  (fj„,_L),  and  thus  the  associated 
listings  do  not  decay.  A  discrete  database  can  be  transformed  into  a  compatible  H  by  setting  an  artificial 
timeout  to  allow  for  decay  (e.g.,  {tin,  tin  +  x)  where  x  is  the  timeout). 

Overlapping  listings  {i.e.,  an  entity  having  more  than  one  active  listing  at  a  time)  are  not  a  concern 
for  our  spam  implementation.  It  is  the  case  for  IP  blacklists  that  an  entity  is  actively  listed  at  most  once, 
and  such  guarantees  are  helpful  in  normalizing  reputation  values.  Nonetheless,  overlapping  listings  can  be 
handled  by  PreSTA,  provided  some  pre-processing  is  performed  over  the  feedbacks. 


4  Spam  Detection  Setup 

In  this  section,  we  describe  the  application  of  the  PreSTA  model  for  the  purpose  of  spam  detection.  Two 
properties  of  spam  and  IP  blacklists  are  well  leveraged  by  PreSTA.  First,  spammers  are  generally  found 
“near”  other  spammers.  Their  identifiers,  IP  addresses,  are  easily  grouped  spatially  due  to  the  hierarchical 
nature  of  IP  address  assignment.  Second,  blacklists  are  a  rich  source  of  easily  accessible  temporal  data. 

It  should  be  emphasized  that  IP  blacklists  are  not  a  prerequisite  to  using  PreSTA  for  spam  detection. 
Any  manner  of  negative  feedback  associating  spamming  and  IP  addresses  is  sufficient.  IP  blacklists,  how¬ 
ever,  are  a  well-regarded  and  generally  trusted  source  of  negative  feedback.  They  are  centrally  maintained 


5 


and  reputation  computed  upon  them  can  be  seen  as  a  good  global  quantifier.  However,  IP  blacklists  do  have 
weaknesses,  and  readers  should  take  care  not  to  associate  these  flaws  to  the  PreSTA  model. 

4.1  Data  Sources 

Blacklist:  To  collect  blacklist  data,  we  subscribe  to  a  popular  blacklist-provider,  Spamhaus  [7].  The 
arrival  and  exit  of  IP  addresses  listed  on  three  Spamhaus  blacklists  (updated  at  thirty-minute  intervals)  were 
recorded  for  the  duration  of  the  experiment^ : 

•  Policy  Block  List  (PBL):  Listing  of  dynamic  IP  addresses  (e.g.,  those  provided  by  large  ISPs 
such  as  Comcast  or  Verizon),  from  which  mail  should  never  originate,  on  principle. 

•  Spamhaus  Block  List  (SBL):  Manually-maintained  listing  of  IPs  of  known  spammers  and  spam 
organizations.  Typically  these  are  IPs  mapping  to  dedicated  spam  servers. 

•  Exploits  Block  List  (XBL):  Automated  listing  of  IPs  caught  spamming;  usually  open  proxies  or 
machines  that  have  been  compromised  by  a  hot-net. 

As  the  latter  two  blacklists  contain  IP  addresses  known  to  have  participated  in  spamming,  only  these  are 
used  to  build  reputation.  The  PBL  is  a  preventative  measure  (though  we  do  use  its  entries  when  examining 
blacklist  performance).  The  mechanism  by  which  a  blacklist  entry  occurs,  be  it  accurate  or  otherwise, 
is  beyond  the  scope  of  this  work.  PreSTA  considers  all  negative  feedback  equally,  and  as  such,  is  not 
dependent  on  the  means  by  which  an  IP  becomes  blacklisted.  Some  Spamhaus  blacklists  (PBL  and  SBL)  list 
IP-prefixes  (blocks)  as  opposed  to  individual  IPs,  but  this  is  no  different  than  listing  each  IP  independently. 

Removal  from  the  blacklist  takes  two  forms:  manual  de-listing  and  timed-expiration.  Given  its  rigorous 
human  maintenance,  the  SBL  follows  the  former  format.  The  XBL,  on  the  other  hand,  defaults  to  a  more 
automated  time-to-live  style.  Empirical  evidence  shows  the  bulk  of  such  listings  expire  5-days  after  their 
appearance  (see  Eig.  1).  However,  in  the  case  a  blacklisted  party  can  demonstrate  its  innocence  or  show  the 
spam-generating  exploit  has  been  patched,  manual  removal  is  also  an  option  for  the  XBE.  Manual  de-listings 
can  complicate  the  calculation  of  MAX_REP,  but  as  we  will  show,  worst  case  spamming  behaviors  are  rarely 
realized,  permitting  strong  normalization. 

AS  Mappings:  Eor  the  purpose  of  mapping  an  IP  address  to  the  Autonomous  System(s)  that  homes  or 
originates  it,  we  use  the  reports  generated  by  CAIDA  [2].  These  are  compiled  from  Route  Views  [8]  data 
and  are  essentially  a  snapshot  of  the  BGP  routing  table. 

E-mail  Set:  Eor  testing  purposes,  we  procured  approximately  31  million  email  headers  collected  at  the 
University  of  Pennsylvania  engineering  email  servers  between  8/1/2009  and  12/31/2009.  The  mail  servers 
host  over  6,100  accounts,  of  which  approximately  5,500  serve  human-users,  while  the  remaining  are  for 
various  administrative  and  school  uses  (i.e.,  courses,  aliases,  lists,  etc.).  Pertinent  information  included  only 
the  time-stamp  of  receipt  and  the  connecting  email  server’s  IP  address  [14]. 

A  considerable  number  of  emails  (2.8  million)  in  our  data-set  were  both  sent  and  received  within  the 
university  network.  Such  emails  were  not  considered  in  our  analysis.  Many  intra-network  messages  are 
the  result  of  list-serves  and  aliasing,  and  by  removing  them,  only  externally  aniving  emails  are  considered. 
Our  working  set  is  further  reduced  to  6.1  million  emails  when  analysis  is  conducted  “above  the  blacklist”  - 
focusing  only  on  those  mails  passing  IP  blacklists. 

To  categorize  email  in  our  data-set  as  either  spam  or  ham  (not  spam),  we  were  provided  a  Proofpoint  [6] 
score  (in  addition  to  the  aforementioned  headers).  Proofpoint  is  a  commercial  spam  detection  service  em¬ 
ployed  by  the  University  whose  detection  methods  are  known  to  include  proprietary  filtering  and  Bayesian 

^Although  blacklist  data  is  pulled  every  30  minutes,  a  time-stamp  is  provided  to  identify  precisely  when  a  new  IP  is  listed. 
However,  no  time-stamp  is  available  for  de-listing,  and  the  timing  of  this  event  is  inferred. 
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content  analysis  [28]  similar  to  that  employed  by  SpamAssassin  [1],  an  open  source  alternative.  Proofpoint 
claims  99.8%  accuracy  with  a  low  false-positive  rate.  Given  no  other  consistent  scoring  metric  and  a  lack  of 
access  to  the  original  email  bodies,  we  use  the  Proofpoint  score  as  the  control  classifier  for  our  analysis. 

A  fair  question  is  then,  “If  text-based  analysis  is  so  accurate,  what  benefit  is  PreSTA?”  The  gains  are 
three-fold:  (1)  Text-based  analysis  exposes  private  email  to  third  party  sources  (often  such  services  are  web- 
based,  as  opposed  to  local  software).  (2)  It  is  far  easier  for  spammers  to  change  email  bodies  than  their 
mail-server’s  IP  address.  And,  (3)  PreSTA  can  be  implemented  as  a  lightweight  pre-processor  (perhaps 
in-network)  to  such  text-based  filtering,  reducing  the  overhead  of  computationally  expensive  analysis. 

4.2  Temporal  Properties 

PreSTA  leverages  the  temporal  properties  of  IP  blacklists  by  aggregating  the  de-listing  and  re-listing  rates 
of  blacklist  entries.  Fig.  1  displays  our  analysis  of  those  two  statistics.  Of  IP-addresses  de-listed  during  our 
experiment,  26%  were  re-listed  within  10  days.  Overall,  47%  of  such  IPs  were  re-listed  within  10  weeks, 
and  it  is  precisely  such  statistics  that  motivate  PreSTA ’s  use  of  temporal  data 

Given  that  IP  addresses  are  frequently  re-listed,  we  examined  the  rate  at  which  de-listing  occurs.  Empir¬ 
ical  analysis  shows  that  80%  of  XBL  entries  were  de-listed  at,  or  very  close  to,  5  days  after  their  entry  (see 
Fig.  1).  Even  so,  this  5-day  interval  is  not  fixed.  Despife  a  non-exacf  expiration,  MAX_REP  is  well  compufed 
using  d  =  5  (days)  as  a  lower-bound.  Raw  repufafion  values  rarely  exceeded  fhe  calculafed  MAX_REP  (less 
fhan  0.01%  of  fhe  fime).  Moreover,  fhe  worsf-case  repufafion  is  achieved  when  fhe  fime  fo  re-lisfing  is  effec¬ 
tively  zero.  The  shorfesf  observed  re-lisfing  inferval  was  approximafely  6  minufes,  and  re-lisfings  infervals 
on  fhis  order  of  duration  are  exfremely  rare. 

Manually  mainfained,  fhe  SBE  has  no  consisfenf  lisfing  lengfh,  and  compufing  MAX_REP  cannof  be 
performed  using  fhe  same  analysis.  Instead,  fhe  manual  maintenance  of  fhe  blacklisf  can  be  used  as  a  faclor 
in  repufafion.  For  an  IP  fo  be  de-lisfed,  if  musf  be  verified  as  a  non-spamming  address.  Thus,  fhere  is  no 
reason  fo  decay  enfries  as  fhey  exif  fhe  lisf^.  Thaf  is,  Vfo«t,  decay{tout)  =  0,  buf  decay(_L)  is  still  1.  In  such 
circumsfances,  fhe  MAX_REP  value  for  such  IPs  is  compufed  as  I  (i.e.,  fhe  IP  address  is  currenfly  lisfed). 

reader’s  intuition  may  be,  “If  SBL  entries  are  not  decayed  and  the  decay  output  simply  echoes  the  SBL  status,  why  is  the 
SBL  needed  for  reputation?”  The  answer  is  spatial:  SBL  and  XBL  listed  IPs  may  reside  inside  the  same  spatial  grouping,  and  SBL 
listings  are  representative  of  malicious  behavior  which  should  lower  the  reputation  for  that  grouping. 
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AS-29385  IP  Space  (256-IP  blocks) 
(Vertical  bar  delineates  non-contiguous  address  space) 


Figure  2:  Behavioral  Variance  within  an  AS 


By  adjusting  the  decay{)  function  in  this  way,  reputations’  of  SBL  IPs  are  based  solely  on  spatial 
properties.  Our  reputation  model  allows  for  such  flexibility  when  the  feedback  database  permits.  In  a 
similar  way,  one  can  focus  solely  on  temporal  properties.  No  matter  the  case,  all  such  reputations  (of 
varying  groupings  and  decays)  can  be  used  in  combination  (see  Sec.  5.7). 

4.3  Spatial  Groupings 

The  hierarchical  nature  of  IP  address  assignment  provides  natural  spatial  groupings  for  use  by  PreSTA. 
Starting  at  the  lowest  level,  a  local  router  or  DHCP  service  assigns  IP  addresses  to  individual  machines.  The 
selection  pool  is  likely,  but  not  necessarily,  well  bounded  to  a  subnet  (i.e.,  a  /24  or  /16).  In  turn,  these  routers 
operate  within  an  ISP/AS,  and  ISP/AS  get  their  allocations  from  Regional  Internet  Registries  (RIRs),  whose 
space  is  delegated  from  the  Internet  Assigned  Number  Authority  [4]  (lANA).  A  clear  hierarchy  exists,  and 
at  each  level,  a  unique  reputation  can  be  applied. 

We  focus  our  groupings  at  three  levels  when  considering  an  IP  address:  (I)  the  IP  addresses  itself,  (2) 
the  768-IP  block  membership  (approximating  the  subnet),  and  (3)  the  AS  that  homes/originates  the  IP.  It 
may  be  the  case  that  an  IP  address  is  multi-homed  -  advertised  by  two  or  more  ASes.  In  such  situations, 
where  multiple  groupings  occur  for  a  single  grouping  function,  a  data-specific  choice  should  be  made.  We 
consider  only  the  highest  reputation  among  the  advertising  ASes,  a  choice  we  discuss  further  in  Sec.  5.5. 

In  addition  to  the  address  space  hierarchy,  other  spatial  relationships  could  be  leveraged  to  form  group¬ 
ings.  The  social  network  of  email  receivers  has  been  proposed  as  an  effective  means  of  spam  filtering,  as 
well  as  network  distance  [11,  13].  In  [15],  geographic  distance  was  used  as  a  classifier.  Although  not  inves¬ 
tigated  herein,  groups  based  on  social,  routing,  or  geographic  distance  could  be  reasonable,  and  the  addition 
of  these  features  is  an  area  of  future  research. 

Despite  its  easily  partitioned  nature,  it  has  yet  to  be  shown  that  the  IP  assignment  hierarchy  provides 
relevant  groupings.  Previous  work  and  anecdotal  evidence  suggest  that  AS-number  is  one  of  the  strongest 
identifiers  of  spammers.  Indeed,  entire  AS/ISPs,  such  as  McColo  [21]  and  3FN  [22],  have  been  shut  down 
as  a  result  of  their  malicious  nature.  Moreover,  in  [15],  AS-level  identifiers  were  used  as  a  reliable  indicator 
of  spamming  hosts,  after  observation  indicated  only  20  ASes  host  42%  of  spamming  IPs. 

At  the  subnet  level,  we  found  that  groupings  of  768  IP-addresses  {i.e.,  three  adjacent  /24s)  well  contain 
malicious  activity  (see  Sec.  5.5  for  details).  As  seen  in  Fig.  2,  we  visualized  /24  blocks  of  address  space  for 
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Figure  3:  PreSTA  Spam  Detection  Architecture 


an  ISP  in  Uzbekistan  by  their  quantity  of  XBL  listings.  Clearly,  there  is  strong  variance  across  the  address 
space  -  some  regions  are  highly  listed  while  others  have  no  listings  whatsoever.  The  AS -level  reputation  of 
this  ISP  is  comparatively  poor  due  to  the  quantity  of  listings,  but  within  the  address  space,  certain  block- 
level  reputations  are  ideal.  This  suggests  that  using  AS -level  reputation  alone  may  be  inappropriate  because 
it  is  too  broad  a  metric.  Alternatively,  it  could  be  that  there  is  no  mail  originating  from  these  blocks,  but  as 
described  in  Sec.  5.7,  we  have  automated  ways  of  dealing  with  such  contradictory  information. 

Finally,  using  a  grouping  function  that  singularly  groups  entities  effectively  removes  spatial  relevance 
from  reputation  computation.  Intuitively,  the  reputation  of  a  single  IP  address  should  be  considered  because 
many  mail  servers  use  static  addresses.  However,  the  often  dynamic  nature  of  address  assignment  implies 
that  unique  IP  addresses  are  not  singular  groupings,  but  rather,  could  represent  many  different  machines 
over  time.  A  recent  study  reported  that  the  percentage  of  dynamically  assigned  IP  addresses'^  on  the  Internet 
is  substantial  and  that  96%  of  mail  servers  using  dynamic  IPs  send  spam  almost  exclusively  [33]. 

5  Spam  Detection  Implementation 

In  this  section  we  describe  the  implementation  of  PreSTA  for  spam  detection.  The  implementation  is 
designed  with  three  primary  goals.  It  should  produce  a  classifier  that  is  (1)  lightweight,  (2)  capable  of 
detecting  a  large  quantity  of  spam,  and  (3)  do  so  with  a  low  false-positive  rate.  We  justify  our  design 
decisions  with  respect  to  these  goals.  Further,  we  discuss  the  practical  concerns  of  such  an  implementation. 

At  a  high-level,  our  work-flow  begins  when  an  email  is  received  and  the  connecting  IP  address  and 
time-stamp  of  receipt  is  recorded.  Assuming  the  IP  is  not  actively  blacklisted,  our  spatio-temporal  approach 
is  brought  to  bear.  The  IP  is  mapped  to  its  respective  spatial  groupings:  itself,  its  subnet,  and  its  originating 
AS(es).  Reputations  are  calculated  at  each  granularity,  and  these  component  reputations  are  supplied  as 
input  to  a  machine-learning  classifier  trained  over  previous  email.  The  output  is  a  binary  ham/spam  label 
along  with  each  of  the  three  component  reputations  -  all  of  which  may  be  of  use  to  a  client  application.  We 
now  describe  this  procedure  in  detail,  and  present  a  visual  reference  of  the  PreSTA  work-flow  in  Fig.  3. 

''Recall  that  Spamhaus’  PEL  blacklist  is  essentially  a  listing  of  dynamic  IP  addresses.  We  note  it  is  constructed  mainly  using 
ISP-provided  data,  and  as  such,  is  far  from  a  complete  listing. 
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5.1  Traditional  Blacklists 


In  Sec.  4.1  the  various  Spamhaus  blacklists  were  introduced.  They  not  only  provide  the  basis  on  which 
reputations  are  built,  but  in  an  implementation  of  PreSTA,  it  is  natural  to  apply  them  as  intended  -  to  label 
emails  originating  from  currently  listed  IPs  as  spam.  When  applied  to  the  email  data-set,  the  Spamhaus 
blacklists  (PEL  included)  captured  91.0%  of  spam  with  a  0.74%  false-positive  rate.  Interestingly,  this  is 
somewhat  higher  than  previous  published  statistics^  [19].  Had  we  chosen  not  to  exclude  the  intra-network 
emails  from  analysis,  the  blacklists  would  have  captured  a  similar  90.9%  of  spam  emails  with  a  much- 
reduced  0.46%  false-positive  rate.  The  exclusion  of  such  emails,  while  inflating  false-positive  rates,  allows 
us  to  concentrate  only  on  the  more  interesting  set  of  externally-received  emails  and  not  bias  our  results. 

The  usage  of  blacklists  (independent  of  spatio-temporal  properties),  enables  fast  detection  of  a  large 
portion  of  spam  emails  with  minimal  time  and  space  requirements.  In  the  implementation  of  PreSTA, 
we  cache  the  current  blacklists  to  improve  look-up  speed,  requiring  roughly  100MB  of  space;  a  reasonable 
requirement  for  most  email  servers.  Further,  a  local  store  of  the  blacklist  is  needed  for  PreSTA  calculation 
because  forms  the  basis  of  the  historical  database. 

5.2  Historical  Database 

Before  reputation  can  be  calculated,  a  historical  feedback  database  must  be  in  place.  As  described,  we 
retrieve  the  Spamhaus  blacklists  at  30-minute  intervals.  The  di  f  f  is  calculated  between  consecutive  copies 
and  time-stamped  entries/exits  are  written  to  a  database.  When  a  new  listing  appears,  we  permanently  record 
the  spatial  groups  (IP,  subnet,  and  AS(es))  that  IP  is  a  member  of.  For  example,  if  IP  i  was  blacklisted  as  a 
member  of  AS  a,  that  entry  will  always  be  a  part  of  a’s  blacklist  history. 

We  found  that  roughly  1GB  of  space  is  sufficient  to  store  one  month’s  blacklist  history  (the  XBL  has 
1.0-1. 5  million  IPs  turn  over  on  a  daily  basis).  Fortunately,  an  extensive  history  is  not  required  given  the 
exponential  decay{)  function^.  For  example,  given  a  10-day  half-life,  a  3-month  old  XBL  entry  contributes 
0.6%  the  weight  of  an  active  listing.  Lengthy  histories  offer  diminishing  returns.  To  save  space,  one  should 
discard  records  incapable  of  contributing  statistical  significance.  Further,  such  removal  saves  time  because 
the  smaller  the  set  hist{)  returns  the  fewer  values  which  must  be  processed  by  rawjrep{). 

5 .3  Grouping  F unctions 

Given  an  entity  (IP  address)  for  which  to  calculate  reputation,  we  must  determine  to  which  groups  this  entity 
belongs  through  the  use  of  our  three  grouping  functions: 

•  IP  Function:  An  IP  is  a  group  in  and  of  itself,  so  such  a  grouping  function  min~ors  its  input.  As 
noted  earlier,  singular  groups  are  interesting  because  over  time,  an  IP  may  have  multiple  inhabitants. 

•  Subnet  Function:  IP  subnet  boundaries  are  not  publicly  available.  Instead,  an  estimate  considers 
blocks  of  IP  addresses  (we  use  the  terms  “subnet-level”  and  “block-level”  interchangeably).  IP  space 
is  partitioned  into  /24s  (256  IP  segments),  and  an  IP’s  block  grouping  consists  of  the  segment  in 
which  it  resides  as  well  as  the  segment  on  either  side;  768  addresses  per  block.  Thus,  block  groupings 
overlap  in  the  address  space,  and  a  single  IP  input  returns  one  block  of  IPs  (three  /24s).  Although 
these  subnet  estimations  may  overflow  known  AS  boundaries,  these  naive  blocks  prove  effective. 

•  AS  Function:  Mapping  an  IP  to  its  parent  AS(es)  requires  CAIDA  [2]  and  Route  Views  [8]  data. 
We  note  that  some  AS  boundaries  overlap  in  address  space  and  some  portions  of  that  space  (i.e., 
unallocated  portions)  have  no  resident  AS  whatsoever.  An  IP  can  be  homed  by  any  number  of  ASes, 

^Our  analysis  of  blacklist  performance  is  from  a  single-perspective,  and  therefore  may  not  speak  to  global  blacklist  effectiveness. 
®This  minimal  history  requirement  was  of  benefit  to  our  own  study.  Reputations  must  warm-up  before  their  use  is  appropriate. 
Indeed,  our  collection  of  blacklist  data  began  in  5/2009,  three  months  before  our  first  classifications. 
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including  none  at  all,  the  technical  considerations  of  which  are  addressed  in  Sec.  5.5.  The  function’s 
output  is  all  the  IPs  homed  by  an  AS(es)  in  which  the  input  IP  is  a  member.  Each  returned  IP  is  tagged 
with  the  parent  AS,  so  a  well-defined  subset  of  the  output  can  be  chosen. 

5.4  Decay  Function 

The  decay  function  controls  the  extent  to  which  temporal  proximity  factors  into  reputation.  This  is  config¬ 
ured  via  the  half-life  parameter,  h.  If  h  is  too  small,  reputations  will  decay  rapidly  and  provide  little  benefit 
over  using  blacklists  alone.  Too  large  an  h  can  cause  an  increase  in  false  positives  due  to  stale  information. 

A  good  half-life  should  maximize  the  difference  between  the  reputation  of  spam  and  ham  email,  and  to 
arrive  at  a  reasonable  value  for  h,  we  analyzed  a  set  of  emails/reputations  pre-dating  our  evaluation  period. 
By  plotting  the  reputation-CDF  for  both  spam  and  ham  email,  we  sought  a  value  for  h  that  maximized  the 
area  between  the  curves.  In  Fig.  4  we  present  the  calculations  from  these  experiments.  We  found  /i  =  10 
(days)  to  optimal  and  therefore  use  this  value  in  our  spam  application^ .  With  the  half-life  established,  and 
having  previously  chosen  d  =  5  (days),  we  calculate  MAX_REP=  4.14. 

As  described  previously,  we  actually  employ  two  separate  decay {)  functions  depending  on  where  a 
listing  appeared,  either  on  the  SBF  or  the  XBF.  Manually  maintained,  we  do  not  decay  de-listing  for  the 
SBF,  but  the  XBF  is  decayed  using  the  aforementioned  10-day  half-life.  In  order  to  use  both  listings 
in  combination,  we  apply  a  flag  to  each  time  pair  returned  by  hist{)  dependent  on  which  blacklist  they 
originated  from,  allowing  us  to  apply  the  appropriate  decay  function. 

5.5  Reputation  Calculation 

Given  the  decay  function  (Sec.  5.4),  output  (sets  of  IP  addresses)  of  the  three  grouping  functions  (Sec.  5.3), 
and  the  feedback  database  (Sec.  5.2),  reputation  may  now  be  calculated.  Valuation  is  performed  at  each 
granularity;  three  reputation  values  are  returned.  Calculation  closely  follows  as  described  in  Sec.  3. 

Calculation  of  IP-level  and  subnet-level  reputation  is  straightforward  per  the  reputation  model  with 
sizei)  =  1  and  size{)  =  768,  respectively.  The  particulars  of  AS-level  calculation  are  more  interesting. 
An  IP  may  be  a  member  of  any  quantity  of  ASes,  including  none  at  all.  If  an  IP  is  multi-homed,  we  make 
the  conservative  choice  by  selecting  the  most  reputable  AS  value  as  the  AS-level  reputation.  Those  IPs 
mapping  to  no  AS  form  their  own  group,  and  we  designate  the  reputation  for  this  group  as  0  because,  in 
general,  unallocated  space  is  only  used  for  malicious  activity.  In  this  spatial  grouping,  sizeQ  is  not  constant 
over  time.  Instead,  magnitudes  are  pre-computed  for  all  AS  using  CAIDA  data  and  updated  as  BGP  routes 
change.  Only  unique  originating  IPs  are  considered  (blocks  often  overlap  to  support  traffic  engineering). 

’Although  we  found  it  unnecessary,  h  could  he  optimized  on  an  interval  basis,  much  like  re-training  a  classifier.  However,  our 
experiments  showed  minor  variations  of  the  parameter  to  he  inconsequential. 
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5.6  Calculation  Optimizations 

To  be  lightweight,  our  system  must  calculate  reputation  efficiently.  It  should  not  significantly  slow  email 
delivery  (latency),  and  it  should  handle  heavy  email  loads  (bandwidth).  We  now  describe  caching  strategies 
and  other  techniques  in  support  of  these  goals: 

•  AS  Value  Caching:  Reputations  for  all  ASes  are  periodically  recalculated  off-line.  These  calcu¬ 
lations  are  (relatively)  slow  given  their  hist{)  calls  return  large  sets. 

•  Block/ip  Value  Caching:  Similarly,  block  and  IP  reputations  can  be  cached  after  the  first  cache 
miss.  Cache  hit  rates  are  expected  to  be  high  because  (1)  an  email  with  multiple  recipients  {i.e., 
a  carbon  copy)  is  received  multiple  times  but  with  the  same  source  IP  address,  and  (2)  source  IP 
addresses  are  non-uniformly  distributed.  For  the  6.1  million  (non-Penn,  non-blacklisted)  emails  in 
our  working  data-set,  there  are  364k  unique  IP  senders  and  I76k  unique  sender  ‘blocks.’ 

•  Cache  Consistency:  Caches  at  all  levels  need  to  be  cleared  when  the  blacklists  are  updated  (every 
30  minutes),  to  avoid  inconsistencies  involving  the  arrival  of  new  listings.  As  far  as  time-decay  is 
concerned,  a  discrepancy  of  up  to  30  minutes  is  inconsequential  when  considering  a  10-day  half-life. 

•  Whitelisting:  There  is  no  reason  to  calculate  reputation  in  trusted  IP  addresses,  such  as  one’s 
own  server.  Of  course,  whitelists  could  also  be  utilized  in  a  feedback  loop  to  alleviate  false-positives 
stemming  from  those  entities  whose  emails  are  consistently  misclassihed. 

Using  these  optimizations,  our  PreSTA  implementation  is  capable  of  scoring  500k  emails  an  hour,  with 
average  email  latencies  well  under  a  second.  Latency  and  bandwidth  are  minimal  concerns.  Instead,  it 
is  the  off-line  processing  supporting  this  scoring  which  is  the  biggest  resource  consumer.  Even  so,  our 
implementation  is  comfortably  handled  by  a  commodity  machine  and  could  easily  run  adjacent  to  an  email 
server.  Pertinent  implementation  statistics,  such  as  cache  performance,  are  available  in  Sec.  6.4. 

5.7  Reputation  Classification 

Extraction  of  a  binary  classification  (i.e.,  spam  or  ham)  is  based  on  a  threshold  strategy.  Emails  valuated 
above  the  thi'eshold  are  considered  ham,  and  those  below  are  considered  spam.  Einding  an  appropriate 
threshold  can  be  difficult,  especially  as  dimensionality  grows,  as  is  the  case  when  classifying  multiple  repu¬ 
tation  values.  Eurther,  a  fixed  threshold  is  insufficient  due  to  temporal  fluctuations;  as  large  groups  (botnets) 
of  spamming  IPs  arise  and  fall  over  time,  and  the  distinction  between  good  and  bad  may  shift. 

Support  vector  machines  (SVM)  [17]  are  employed  to  determine  thresholds.  SVM  is  a  form  of  su¬ 
pervised  learning  that  provides  a  simple  and  effective  means  to  classify  multiple  reputation  values.  The 
algorithm  maps  reputation  triples  (a  feature  for  each  spatial  dimension)  from  an  email  training  set  into  3- 
dimensional  space.  It  then  determines  the  surface  (threshold)  that  best  divides  spam  and  ham  data-points 
based  on  the  training  labels.  This  same  threshold  can  then  be  applied  during  classification.  The  SVM  routine 
can  be  tuned  via  a  cost  metric  which  is  correlated  to  the  eventual  false-positive  rate  of  the  classifier. 

The  classifier  is  adjusted  (re-trained)  every  4  days  to  handle  dynamism.  A  subset  of  emails  received  in 
the  previous  4  days  are  trained  upon,  and  the  resulting  classifier  is  used  for  the  next  4  day  interval.  The 
affect  of  different  training  periods  has  not  been  extensively  studied.  Clearly,  large  periods  are  not  desired; 
the  reputation  of  distant  emails  may  not  speak  to  the  classification  of  more  current  ones.  Too  short  a  period 
is  equally  poor  because  it  requires  extensive  resources  to  re-train  so  frequently.  We  believe  4-day  re-training 
is  a  good  compromise.  However,  the  re-training  period  need  not  be  fixed,  and  future  work  will  explore 
re-training  rates  that  adjust  based  on  various  environmental  factors  at  the  email  server. 

At  each  re-training,  we  used  10,000  emails  (5%  of  the  non-Penn,  non-blacklisted  email  received  every 
4  days),  and  labeled  emails  as  spam/ham  based  on  the  Proofpoint  score.  In  a  more  general  use  case,  there 
would  be  some  form  of  client  feedback  correlated  across  many  accounts  that  can  classify  spam  post-delivery 
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Figure  5:  XBL  Size  Relative  to  Global  Reputation 


and  train  various  spam  detectors.  Since  we  do  not  have  access  to  such  user  behavior,  correlation  statistics, 
or  any  external  spam  filters,  the  provided  Proofpoint  values  are  assumed. 

Post-training,  the  false-positive  (FP)  rate  of  the  classifier  can  be  estimafed  by  measuring  the  error  over 
the  training  set  (assuming  one  does  not  over-fit  the  training  data).  The  estimated  FP-rate  is  a  good  indicator 
of  the  true  FP-rate,  and  the  SVM  cost  parameter  can  be  adjusted  to  tune  the  expected  FP-rate.  All  classifier 
statistics  and  graphs  hereafter  were  produced  with  a  0.5%  tolerance  for  false-positives  (over  the  classification 
set),  as  this  simplifies  presentation.  We  believe  0.5%  is  a  reasonable  setting  given  that  blacklists  are  widely 
accepted  and  achieved  a  0.74%  FP-rate  over  the  same  dataset.  Additionally,  these  rates  are  somewhat 
inflated  given  our  decision  to  exclude  intra-network  emails,  which  are  unlikely  to  contribute  false-positives 
(the  blacklist  FP-rate  reduced  to  0.46%  with  their  inclusion).  In  Sec.  6.5,  the  trade-off  between  the  FP-rate 
and  spam  blockage  is  examined  in  greater  depth. 

6  Empirical  Results 

We  begin  our  PreSTA  spam  detection  analysis  by  examining  the  component  reputations  individually.  From 
there,  two  case  studies  will  exemplify  how  PreSTA  can  produce  metrics  outperforming  traditional  blacklists 
in  both  spatial  and  temporal  dimensions.  Finally,  we  examine  the  effectiveness  of  the  full-fledged  spam  filter: 
For  each  mail  in  our  data-set,  reputation  metrics  are  calculated  and  email  is  valuated  in  a  way  that  would 
mimic  our  PreSTA  implementation  on  a  production  email  server,  complete  with  re-training  and  caching. 

6.1  Blacklist  Relationship 

In  examining  how  our  reputations  quantify  behavior,  we  began  with  a  simple  intuition:  one  would  expect 
to  see  a  clear  push-pull  relationship  between  an  entities  reputation  and  the  number  of  corresponding  entries 
on  the  blacklist.  To  confirm,  we  graphed  the  size  of  the  XBL  blacklist^  over  time  and  compared  this  to 
the  average  reputation  of  all  ASes.  Results  are  presented  in  Fig.  5.  We  observe  an  inverse  relationship, 
confirming  our  expectations.  When  the  number  of  listings  dips,  reputation  increases  -  and  vice  versa. 
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(a)  IP  Reputation 


(b)  Block  Reputation 


(c)  ASN  Reputation 


Figure  6:  CDFs  of  Component  Reputations 


6.2  Component  Reputation  Analysis 

In  order  for  component  reputations  (IP,  block,  and  AS)  to  be  useful  in  spam  detection  they  must  be  behavior 
predictive.  That  is,  the  reputations  associated  to  ham  emails  should  exceed  those  of  spam  emails.  This 
relationship  is  visualized  in  the  cumulative-distribution-functions  (CDFs)  of  Fig.  6.  We  observe  that  all 
component  reputations  behave  as  expected.  Fig.  6  also  shows  the  benefit  of  using  multiple  spatial  groupings. 
While  nearly  90%  of  spam  emails  come  from  IPs  that  had  ideal  reputation  {i.e.,  a  reputation  of  I)  at  the  time 
of  receipt,  this  is  true  for  just  46%  of  blocks,  and  only  3%  of  AS. 

The  CDFs  of  Fig.  6  imply  that  each  component  reputation  is,  in  and  of  itself,  a  metric  capable  of  classi¬ 
fying  some  quantity  of  spam.  However,  it  is  desirable  to  show  that  each  granularity  captures  unique  spam, 
so  that  the  combination  of  multiple  reputations  can  produce  a  higher-order  classifier  of  greafer  accuracy. 
In  Fig.  7,  fhe  effectiveness  of  each  componenf  repufafion  is  presenfed.  The  percenfage  of  spam  caughf  is 
“above  fhe  blacklisf,”  or  more  precisely,  fhe  percenfage  of  spam  well-classified  by  fhe  repufafion  value  fhaf 
was  nol  identified  by  fhe  blacklisf  alone.  Given  fhe  inclusion  of  fradifional  blacklisf  filfering,  fhe  primary 
concern  is  fhose  emails  fhaf  are  nol  actively  listed. 

On  fhe  average,  PreSTA  is  able  fo  capfure  25.7%  of  spam  emails  nol  caughf  by  fradifional  blacklists. 

*The  XBL  is  the  driving  force  behind  reputation.  The  SBL  is  also  a  contributor,  but  is  orders  of  magnitude  smaller. 
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Figure  8:  Single  IP  Behavior  w.r.t.  Blacklisting 


Crucially,  the  combined  performance  (the  top  line  of  Fig.  7),  exceeds  that  of  any  individual  component,  so 
each  spatial  grouping  catches  spam  the  others  do  not. 

We  are  also  interested  in  determining  which  grouping  provides  the  best  classification.  AS-level  reputa¬ 
tion  is  the  most  stable  of  the  components,  individually  capable  of  classifying  an  additional  10-15%  of  spam 
above  the  blacklist.  However,  we  observe  that  during  periods  of  increased  PreSTA  performance,  it  is  often 
the  block  and  IP  levels  that  make  significant  contributions.  This  is  intuitive;  AS-level  thresholding  must  be 
conservative.  Given  their  large  size,  ASes  have  relatively  stable  reputations.  Thus,  the  classification  of  a 
single  reputation  value  may  effectively  make  the  spam/ham  determination  for  many  thousand  emails  -  and 
could  result  in  an  unacceptable  increase  in  the  FP-rate.  Meanwhile,  the  cost  associated  with  a  mis-prediction 
is  far  less  for  block  and  IP  groupings,  permitting  more  aggressive/speculative  thr'esholds. 

These  results  suggest  that  considering  more  spatial  dimensions  can  increase  performance,  that  is,  when 
there  are  non-overlapping  classifications.  However,  there  are  diminishing  returns.  Each  additional  compo¬ 
nent  reputation  requires  increased  resources  for  valuation  and  classification.  An  application  should  seek  a 
minimal  set  of  dimensions  to  best  represent  and  classify  its  data. 

6.3  Case  Studies 

Two  case  studies  are  exemplary  of  the  types  of  spam  behavior  able  to  evade  blacklists,  yet  captured  via 
PreSTA.  First,  Fig.  8  shows  the  temporal  sending  patterns  of  a  single  spamming  IP  address.  This  IP 
was  blacklisted  twice  during  the  course  of  the  study  (as  indicated  by  shaded  regions),  and  the  time  be¬ 
tween  listings  was  small  (roughly  2  days).  The  controller  of  this  IP  address  likely  used  blacklist  counter¬ 
intelligence  [26]  to  increase  the  likelihood  that  spam  would  be  delivered.  Notice  that  no  spam  was  observed 
when  the  IP  was  actively  blacklisted,  but  150  spam  emails  were  received  at  other  times. 

Traditional  blacklist  are  reactive,  binary  measures  that  do  not  take  history  into  account.  During  the 
intermittent  period  between  listings,  as  far  as  the  blacklist  is  concerned,  the  spamming  IP  is  an  innocent  one. 
However,  as  shown  in  Fig.  8,  the  IP-level  reputation  metric  compounds  prior  evidence.  Thus,  if  PreSTA 
had  been  in  use,  the  intermittent  influx  of  email  would  have  been  identified  as  spam. 

Secondly,  Fig.  9  visualizes  a  case  sfudy  af  fhe  AS-level  ufilizing  bofh  spatial  and  temporal  dimensions. 
In  fhe  early  sfages  of  our  dafa  collecfion  we  noficed  anomalous  acfivify  occumng  af  a  particular  AS  (AS#- 
12743)^.  Even  when  compared  fo  fhe  ofher  four  worsf  performing  ASes  during  fhe  time  block,  ASN-12743’s 

®PTK-Centertel,  a  major  Polish  mobile  service  provider 
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Figure  9:  Temporal  Shift  within  Spatial  Grouping  (AS) 


drop  in  reputation  is  astounding.  Nearly  its  entire  address  space,  some  4,500  addresses,  were  blacklisted  in 
the  course  of  several  days  -  likely  indicative  of  an  aggressive  botnet-based  spam  campaign  -  after  which, 
the  reputation  increases  exponentially  (per  the  half-life),  eventually  returning  to  innocent  levels. 

With  traditional  blacklists,  an  IP  must  actually  send  spam  before  it  can  be  blacklisted.  In  the  ASN- 12743 
case,  this  means  all  4,500  IPs  had  some  window  in  which  to  freely  send  spam.  However,  as  the  IPs  were 
listed  in  mass,  the  reputation  of  the  AS  drops  at  an  alarming  rate,  losing  more  than  50%  of  its  value.  Had 
PreSTA  been  implemented,  the  reputation  of  the  AS  (and  the  blocks  within)  would  have  been  low  enough 
to  classify  mails  sourced  from  the  remainder  of  the  space  as  spam,  mitigating  the  brunt  of  the  attack. 

6.4  Implementation  Performance 

The  previous  case  studies  are  but  two  examples  of  the  way  component  reputations  capture  spamming  behav¬ 
ior  that  evades  traditional  blacklists.  We  now  present  the  results  of  the  simulated  PreSTA  implementation. 

Our  experimental  setup  was  as  follows:  To  best  simulate  the  normal  processing  at  a  mail  server,  we 
assumed  each  email  aiTived  in  the  order  of  the  time-stamp.  The  blacklist  history  and  cached  reputation 
scores  were  regulated  so  that  only  the  knowledge  available  at  the  time  of  arrival  is  used  to  valuate  the  email. 
PreSTA  requires  a  warm-up  period  to  gather  enough  temporal  knowledge  to  process  correctly;  hence, 
historical  blacklist  storage  began  three  months  prior  to  the  hrst  email  being  scored. 

We  were  interested  in  measuring  the  effectiveness  of  the  cache  and  the  latency  of  the  system.  Caching 
was  highly  effective:  56.8%  of  block-level  calculations  are  avoided,  and  43.1%  are  avoided  at  the  IP-level 
(recall  that  all  AS-level  calculations  are  performed  off-line  and  then  cached).  As  such,  the  reputation  of  an 
incoming  email  can  be  calculated  in  nearly  real  time,  with  the  average  email  being  processed  in  fractions  of 
a  second.  Under  typical  conditions,  over  500,000  emails  can  be  scored  in  an  hour. 

Re-training  our  classihers  and  rebuilding  the  AS -cache  are  the  most  time  consumptive  activities.  Fortu¬ 
nately,  hnding  new  classihers  takes  only  minutes  of  work  for  a  10,000  email  training  set,  and  only  needs  to 
be  performed  every  4  days.  Re-training  can  also  be  done  off-line  and  not  affect  current  scoring.  Rebuilding 
the  AS  reputation  cache  must  be  done  every  30  minutes  once  new  blacklist  data  is  available,  but  it  need  not 
delay  current  scoring  as  the  previous  AS-level  reputations  are  still  relevant  (at  most  30  minutes  old). 
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Figure  10:  Blacklist  and  PreSTA  Spam  Blockage 


6.5  Spam  Mitigation  Performance 

The  spam  detection  capabilities  of  PreSTA  are  summarized  in  Fig.  10.  On  average,  93%  of  spam  emails 
are  identified  when  using  our  system  in  conjunction  with  traditional  blacklists.  To  some  this  may  seem 
to  be  only  a  nominal  increase  over  using  blacklists  alone.  However,  the  inset  of  Fig.  10  is  more  intuitive; 
plotting  the  PreSTA  blockage  rate  only  over  those  mails  passing  the  Spamhaus  blacklists  (identical  to  the 
top  line  seen  in  Fig.  7).  We  observe  that  between  20%  and  50%  of  spams  evading  blacklists  can  be  caught  by 
PreSTA  (with  a  25.7%  average).  Had  PreSTA  been  implemented  on  our  university  mail  server,  it  would 
have  caught  650,000  spam  emails  that  evaded  the  Spamhaus  blacklists  over  the  course  of  our  study. 

Most  interestingly,  PreSTA  allows  for  a  consistent  and  steady  state  of  spam  detection.  For  example, 
consider  the  significant  drops  in  blacklist  performance  seen  throughout  our  study  (for  example,  in  late  Au¬ 
gust  2009  and  again  in  mid-November  2009).  PreSTA  is  nearly  unaffected  during  these  periods  and  could 
be  used  as  a  stop-gap  to  variance  in  blacklist  accuracy.  Clearly,  whatever  the  means  of  blacklist  evasion 
was  during  these  periods,  it  was  insufficient  to  evade  PreSTA.  Further,  we  believe  future  data  will  show 
such  dips  and  rises  in  blacklist  performance  to  be  non-anomalous.  Periods  of  high  de-listing  are  likely  to 
be  followed  by  periods  of  high  re-listing  as  spammers  try  to  maximize  the  utility  of  available  IPs.  In  the 
interim,  blacklists  are  likely  to  perform  relatively  poorly,  and  PreSTA  could  aid  in  maintaining  a  consistent 
level  of  spam  blockage.  While  the  blockage-rates  of  the  blacklists  fluctuate  18%  over  the  course  of  our 
study,  PreSTA  is  far  more  consistent,  exhibiting  just  5%  of  variance. 

Ultimately,  the  performance  attainable  by  our  classifier  is  dependenf  on  the  number  of  false-positives 
(FPs)  a  user  is  willing  to  tolerate.  To  this  point,  the  FP-rate  has  been  fixed  at  0.5%  in  order  to  simplify 
discussion.  However,  as  exemplified  in  Fig.  11,  fhe  FP-rafe  is  fune-able  and  strongly  correlates  with  the 
blockage  rate.  The  plot  is  generated  over  a  characteristic  interval  of  email  from  mid-October  2009,  and  is 
akin  to  the  precision/recall  graphs  common  in  machine-learning.  We  remind  readers  that  our  decision  to 
exclude  intra-network  emails  from  our  dataset  (see  Sec.  5.1)  significantly  inflates  the  presented  FP-rates. 
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Figure  11:  Characteristic  FP/Blockage  Trade-Off 


7  Evasion  and  Gamesmanship 

In  order  for  a  PRESTA-based  spam  filtering  approach  to  be  effective,  it  must  be  robust  to  evasion.  Given 
that  we  use  blacklists  as  a  feedback  source,  perhaps  the  most  effective  way  to  avoid  PreSTA  detection  is 
to  avoid  getting  blacklisted  in  the  first  place  (this  is  true  in  any  PreSTA  application  where  one  can  avoid 
negative  feedback).  However,  such  a  technique  is  not  fail-safe;  a  single  evasive  entity  may  still  have  poor 
reputation  at  broader  granularity.  Given  that  negative  feedback  does  exist,  and  an  IP  has  been  blacklisted,  the 
best  recourse  is  patience.  Over  time,  the  weight  of  the  listing  will  decrease  according  to  the  decay  function. 
As  such,  there  is  no  way  to  evade  PreSTA  in  the  temporal  dimension. 

However,  spammers  are  migrant,  and  the  spatial  dimension  affords  greater  opportunities.  While  IP  and 
block  magnitudes  are  fixed,  an  AS  controls  the  number  of  IPs  it  broadcasts.  An  actively  evasive  AS  would 
ensure  its  entire  allocation  is  broadcasted.  More  maliciously,  a  spammer  may  briefly  hijack  IP  space  they 
were  not  allocated  in  order  to  send  spam  from  stolen  IPs.  Such  spectrum  agility  was  shown  by  [25]  to  be 
an  emergent  spamming  technique.  Fortunately,  if  the  hijacked  IP  space  was  not  being  broadcasted  {i.e., 
unallocated),  emails  from  these  IPs  would  map  to  the  special  grouping  “no  AS”,  whose  reputation  is  zero 
(per  Sec.  5.5).  However,  if  the  hijacked  space  was  being  broadcasted  by  a  reputable  AS,  evasion  may  be 
possible.  Fortunately,  [25]  observes  the  use  of  unallocated  space  is  most  prevalent. 

As  a  general  purpose  reputation  engine,  a  sizing  attack  can  be  of  real  concern.  The  entities  being  valuated 
should  not  be  able  to  affect  the  size  of  their  spatial  groupings.  However,  this  attack  is  only  effective  when 
the  group  size  is  non-singular,  and  an  easy  avoidance  technique  is  to  always  include  a  grouping  function 
that  defines  singular  groups.  Further,  an  implementation  should  try  to  assign  persistent  identifiers  to  entities. 
When  identifiers  are  non-persistent,  PreSTA  could  fall  victim  to  a  Sybil  attack  [12]  since  an  entity  could 
evade  negative  feedback  by  simply  changing  identifiers. 

8  Additional  PreSTA  Applications 

PreSTA ’s  applicability  is  broader  than  email  spam  alone,  as  the  spatial  and  temporal  properties  described 
are  inherent  in  a  number  of  domains.  Indeed,  PreSTA  reputation  values  have  already  proven  successful 
in  the  detection  of  vandalism  on  Wikipedia  [32].  Any  edit  which  is  blatantly  unproductive,  offensive,  or 
over-zealous  in  its  removal  of  content  is  said  to  exhibit  vandalism.  Prior  to  [32],  attempts  to  detect  these 
edits  resided  only  in  the  language-processing  domain  [23,  29].  Akin  to  the  Bayesian  filters  of  email  spam, 
these  efforts  also  suffer  from  many  of  the  same  drawbacks  (evade-ability  and  minimal  throughput). 

Recall  our  two  PreSTA  application  criteria.  First,  A  view-able  history  of  dynamic  negative  feedback. 
On  Wikipedia,  a  special  administration  form  of  reversion  called  rollback  permits  the  discovery  of  malicious 
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edits,  fulfilling  this  requirement.  Second,  there  should  be  at  least  one  finite  partitioning  of  the  entities. 
The  authors  of  [32]  find  it  appropriate  to  consider  both  users  and  articles  as  the  entities  involved  in  an 
edit.  In  addition  to  singular  groupings,  these  entities  are  grouped  by  geographical-space  and  topic-space 
{i.e.,  categories),  respectively.  Combining  PreSTA  values  with  other  metadata  features,  [32]  ultimately 
produces  a  classifier  comparable  in  performance  to  natural-language  efforts. 

Other  use-cases  for  PreSTA  are  active  areas  of  research,  and  in  particular,  content-based  access  control 
scenarios  seem  most  ripe  for  exploration.  However,  it  may  be  possible  to  generalize  the  PreSTA  model 
further  by  providing  a  reduction  to  dynamic  trust  management  (DTM)  systems  [10,  31],  which  combine 
trust  management  and  reputation  management  fundamentals  to  make  access  control  decisions  using  only 
partial  information.  Credential  delegation  chains  have  hierarchical  properties  from  which  spatial  groups  can 
be  extracted.  Moreover,  DTM  systems  rely  on  feedback  databases  for  their  reputation  component,  which 
PreSTA  could  easily  leverage.  Future  work  could  formalize  this  reduction,  showing  that  PreSTA  may  be 
applicable  to  an  entire  new  class  of  systems. 

9  Conclusions 

In  this  paper,  we  have  introduced  PreSTA,  a  spatio-temporal  reputation  model,  and  demonstrated  its  ef¬ 
fectiveness  by  using  it  for  spam  detection.  PreSTA  has  proven  capable  with  respect  to  spam,  blocking  up 
to  50%  of  spam  emails  not  caught  by  traditional  blacklists,  and  identifying  93%  of  spam  on  average  when 
used  in  combination.  In  particular,  our  method  succeeded  in  being  a  stop-gap  mechanism  for  periods  of 
low  blacklist  performance.  Our  technique  is  also  scalable  and  able  to  efficiently  handle  production  email 
workload,  at  least  at  the  level  of  a  university  mail  system,  processing  over  500,000  emails  an  hour. 

We  do  not  propose  PreSTA  based  spam  detection  as  a  replacement  for  context-based  analysis  systems. 
However,  we  believe  it  could  be  useful  as  an  intermediate  filter,  perhaps  centrally  maintained  and  queried 
like  DNS-based  IP  blacklists  are  today.  Alternatively,  the  reputation  values  PreSTA  computes  could  be 
used  in  combination  with  other  features  to  produce  classifiers  of  even  greater  accuracy.  With  only  a  small 
amount  of  extra  processing,  PreSTA  was  able  to  turn  a  reactive  blacklist  into  a  predictive  service. 

Further,  we  believe  PreSTA  has  applicability  beyond  spam,  and  recent  related  work  with  Wikipedia 
has  already  shown  this  to  be  the  case.  Any  application  meeting  our  two  criteria  is  a  potential  use  case. 
No  matter  the  application,  PreSTA’s  power  is  derived  from  its  ability  to  take  a  historical  record  of  bad 
behavior  and  produce  from  that  predictive  identifications  of  additional  malicious  entities.  This  is  achieved 
by  analyzing  not  just  an  individual  entity’s  historical  behavior,  but  also  the  histories  of  groups  wherein  the 
entity  resides.  Thus,  in  the  absence  of  entity-specific  dafa,  we  are  able  fo  rely  on  spatial  and  temporal  data 
to  make  a  characterization.  By  combining  reputations  from  varying  granularity,  we  are  able  to  produce 
robust  reputation  values  that,  as  a  result  of  normalization,  are  comparable.  Ultimately,  the  reputations  may 
be  utilized  as  an  effective  means  of  performing  dynamic  access-control  and  mitigating  malicious  behavior, 
two  extremely  relevant  issues  as  service  paradigms  shift  to  more  distributed  architectures. 
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